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ABSTRACT 


Tactical  air  missions  place  a  high  visual  load  on  pilots,  who  must  rapidly  acquire  and 
synthesize  information  from  displays  within  the  cockpit  (near-domain)  and  events  outside  of  the 
cockpit  (far  dom.ain;  Martin-Emerson  &  Wickens,  1997).  The  delivery  of  target-position 
information  is  especially  critical,  as  even  slight  delays  can  have  catastrophic  consequences.  The 
development  of  head-up  (HUD)  and  hel.met-mounted  displays  (HMD),  which  superimpose  near¬ 
domain  elements  onto  the  far-domain  scene  and  provide  visual  cueing  to  target  position,  has 
im.proved  performance  in  target  detection  scenarios  (Fadden,  Ververs,  &  Wickens,  1998;  Osgood, 
Wells,  &  Meador,  1995).  While  helpful,  such  technologies  have  restricted  display  areas,  orfields- 
of-view  (FOV;  Davis,  1 997),  which  limit  their  effectiveness  in  finding  and  designating  targets. 
Recent  research  has  shown  that  spatia'ized  audio  cueing  can  lead  to  increased  detection  rates, 
reductions  in  detection  time,  and  reductions  in  workload  during  visual  search  tasks  (Begault, 

1993;  Bronkhorst,  Veltman,  &  Breda,  1996;  Nelson,  Hettinger,  Cunningham,  Brickman,  Haas,  & 
McKinley,  1998).  However,  research  on  spatialized  auditory  cueing  when  using  HMDs  has 
focused  on  opaque  HMDs  that  present  virtual  environments  as  compared  to  the  richer  visual 
scene  available  via  see-through  HMDs  typically  found  in  operational  settings.  Therefore,  one  goal 
for  the  present  study  was  to  determine  the  benefits  of  combining  auditory  and  visual  cueing  for 
target-designation  performance  during  a  simulated  flight-task  using  a  see-through,  head-coupled 
visual  display.  It  was  expected  that  when  compared  to  non-cued  and  unimodally  cued  conditions, 
multimodal  audio-visual  interfaces  would  lead  to  more  efficient  performance,  particularly  for 
targets  initially  outside  of  the  display’s  field-of-view. 

An  important  consideration  for  implementing  displays  is  their  interaction  with  existing 
interfaces.  Cockpits  have  limited  space  for  information  display,  and  there  is  already  a  high  visual 
demand  that  may  contribute  to  information  overload  and  increase  the  probability  of  pilot  error 
(Reising,  Liggett,  &  Munns,  1999;  Weinstein  &  Wickens  1992).  Combining  visual  cueing  with 
auditory  cueing  might  exacerbate  that  clutter.  A  promising  approach  to  counteracting  clutter  is  the 
use  of  adaptive  interfaces,  in  which  automation  is  employed  to  control  the  delivery  of  information 
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to  pilots  so  that  they  receive  the  right  information  in  the  right  format  at  the  right  time  and  are  not 
otherwise  exposed  to  that  information  (Hettinger,  Cress.  Brickman,  &  Haas,  1996;  Holinagel, 
1988).  This  approach  has  been  shown  to  be  helpful  to  pilots  during  landing  and  navigational  tasks 
(Brickm.an,  Hettinger,  Roe,  Lu,  Repperger,  and  Haas;  1996,  Moroney,  1999).  Accordingly,  a 
second  goal  for  the  present  investigation  was  to  evaluate  the  role  of  adaptive  cueing  interfaces  as 
a  means  of  integrating  visual  and  auditory  displays  for  target  designation.  It  was  anticipated  that 
performance  with  multimodal  adaptive  interfaces,  which  reduce  cockpit  clutter,  would  be  superior 
to  that  associated  with  fixed-format  multimodal  interfaces. 

The  study  was  conducted  at  the  Air  Force  Research  Laboratory's  Synthesized  Immersion 
Research  Environment  (SIRE)  at  Wright-Patterson  Air  Force  Base.  Twelve  experienced  pilots  (1 1 
males  and  1  female;  mean  2652  flight  hours)  serving  at  the  base  participated  in  the  study.  While 
perform.ing  a  simulated  flight  task  in  which  they  were  instructed  to  maintain  a  prescribed  flight 
path,  airspeed,  and  altitude,  pilots  visually  searched  out-the-window  for  ground  and  air  targets. 
The  targets  could  initially  appear  either  within  or  outside  of  the  display  FOV.  Seven  target- 
location  cueing  conditions  were  employed.  They  included  a  non-cueing  control  and  six  cueing 
interfaces.  These  head-coupled  cueing  interfaces  featured  auditory  (headphone-delivered 
spatialized  sound),  visual  (simulated  HMD  symbology),  fixed  multimodal  (simultaneous  auditory 
and  visual),  and  adaptive  multimodal  display  configurations.  In  the  case  of  adaptive  displays,  the 
presentation  of  auditory  and  visual  cues  was  determined  by  whether  a  target  was  positioned 
inside  or  outside  of  the  HMD  FOV.  Thus,  in  the  Adaptive  Auditory  plus  Adaptive  Visual  condition, 
a  target  initially  beyond  the  display  FOV  would  be  cued  aurally.  When  the  visual  display 
encompassed  the  target  in  the  display  area,  the  auditory  cueing  would  be  replaced  by  the  HMD- 
based  visual  cueing  symbology.  In  the  Adaptive  Auditory  plus  Visual  condition,  auditory  cueing 
was  provided  when  a  target  was  present  outside  the  FOV,  while  visually  cueing  remained 
regardless  of  FOV.  In  the  Adaptive  Visual  plus  Auditory  condition,  visual  cueing  was  only 
presented  when  the  target  was  within  the  FOV,  while  auditory  cueing  was  provided  whenever  a 
target  was  present.  All  pilots  participated  in  all  conditions.  In  addition  to  target  designation 
accuracy  and  time,  visual  search  performance  was  evaluated  in  terms  of  pilot  head  motion 
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because  biomechanical  fatigue  is  an  important  factor  in  the  use  of  HMDs  (Davis,  1997).  Flight 
performance  and  NASA-TLX  ratings  were  also  collected  to  assess  the  impact  of  the  several 
cueing  interfaces  on  aircraft  control  and  subjective  workload. 

Designation  accuracy  was  poorest  in  the  non-cueing  and  auditory  display  conditions  in 
comparison  to  the  visual,  fixed  multimodal,  and  adaptive  multimodal  conditions,  which  did  not 
differ  amiong  themselves.  Consequently,  the  non-cueing  and  auditory  conditions  were  not 
considered  in  terms  of  designation  time,  and  the  visual  cueing  condition  served  as  the  principal 
baseline  for  that  measure.  The  addition  of  spatialized  sound  to  visual  cueing  reduced  target 
designation  time  in  comparison  to  the  visual  cue  alone  under  conditions  in  which  targets  were 
most  difficult  to  detect:  ground  targets  initially  outside  of  the  field-of-view.  Such  targets  had 
relatively  low-contrast  with  the  terrain,  making  it  difficult  to  segregate  them  from  the  background, 
as  evidenced  by  the  lower  hit  rates  for  the  ground  as  compared  to  the  air  targets.  The  time 
advantage  for  multimodal  cueing  (both  fixed  and  adaptive)  of  approximately  825msec  is  of 
considerable  practical  significance,  since  even  slight  time  advantages  can  be  critical  for  tactical 
aviation  (Barnes,  1989).  The  failure  to  find  benefits  for  supplementary  auditory  cueing  as 
compared  to  visual  cueing  alone  in  the  other  target  type/FOV  combinations  may  be  attributable  to 
the  limited  resolution  of  the  auditory  cueing  system,  as  well  as  pilots’  reliance  on  visual  displays 
and  their  lack  of  experience  with  spatialized  auditory  cueing. 

In  addition  to  enhancing  visual  search  performance,  multimodal  cueing  had  the  added 
benefits  of  reducing  excessive  head  motion  and  of  lowering  pilots’  workload  by  approximately 
30%  in  comparison  to  the  case  in  which  no  cueing  was  available.  Moreover,  multimodal  cueing 
did  not  interfere  with  the  flight  task;  i.e.  multimodal  cueing  did  not  incur  a  performance-resource 
debt  (cf.  Wickens,  1992).  In  sum,  the  present  study  suggests  that  multimodal  cueing  may 
effectively  aid  target  localization  in  tactical  aviation. 

The  present  investigation  revealed  no  advantage  for  presenting  multimodal  information 
adaptively  over  presenting  it  in  a  fixed  format;  the  benefits  associated  with  multimodal  information 
were  identical  in  both  fixed  and  adaptive  formats.  The  failure  of  the  adaptive  conditions  to  prevail 
over  the  fixed  format  conditions  in  this  study  may  stem  from  the  fact  that  the  adaptive  format  was 
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not  necessary  in  the  flight  scenario  used  here.  Since  the  fixed  muitimoda!  interface  did  not 
interfere  with  the  performance  of  the  concomitant  fiight  tasks,  there  was  no  added  benefit  in 
clutter  reduction  to  be  gained  through  the  adaptive  format.  The  usefulness  of  an  adaptive  format 
in  reducing  the  demands  of  cockpit  clutter  when  presenting  muitimoda!  information  about  target 
localization  might  em.erge  in  a  fiight  scenario  in  which  the  quantity  and  demand  characteristics  of 
near-domain  information  exceed  those  used  in  the  present  investigation. 
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CHAPTER  1 


Introduction 

Summary  of  the  Problem 

One  of  the  most  challenging  of  human  factors  domains  is  tactical  aviation.  The 
complexity  and  dangers  presented  by  such  human-machine  environments  necessitate  the 
development  of  display  technologies  that  optimize  pilot  performance  and  minim.ize  errors. 
Although  recent  developments  in  visual  and  auditory  display  technology  have  enhanced 
performance  in  flight  tasks  (e.g.,  navigation,  target  localization),  there  are  unresolved  issues 
concerning  when  and  how  to  present  c.ntical  information,  especially  given  display  limitations  and 
the  already  information-loaded  cockpit.  The  purpose  of  the  present  study  was  to  evaluate  the 
perform, ance  and  demand  characteristics  associated  with  several  visual  and  audito,^  display 
configurations  (interfaces)  designed  to  aid  pilots  in  locating  targets  (cueing)  during  a  simulated 
mission.  The  cueing  configurations  included  single  (unimodal)  and  combined  (multimodal) 
auditory  and  visual  displays,  as  well  as  adaptive  display  integration,  in  which  the  modality  of 
presentation  varied  with  situational  demands. 

The  effectiveness  of  the  interfaces  was  assessed  via  measures  of  target  search  and  flight 
performance,  as  well  as  pilot  head  motion  and  perceived  workload.  It  was  expected  that  this 
research  would  provide  guidance  for  the  implementation  of  multimodal  displays  and  for  the  utility 
of  adaptive  display  integration,  across  a  range  of  tasks,  both  within  and  beyond  tactical  aviation 
environments. 

Information  Load  of  the  Tactical  Cockpit 

Information  Acquisition.  Tactical  air  missions  impose  high  visual  demands  on  pilots, 
particularly  in  single-seat  fighter  aircraft.  In  addition  to  the  maintenance  of  proper  course, 
altitude,  and  airspeed,  pilots  must  perform  several  other  functions,  including  weapons  and  fuel 
management,  radio  communication,  and  monitoring  the  environment  for  potential  targets  and 
threats.  Successful  performance  of  these  tasks  depends  on  pilots’  ability  to  acquire  and 


1 


synthesize  information  from  different  sources.  These  include  near-domain  information  about 
system  functions  (presented  by  cockpit  instrumentation)  and  far-domain  (i.e.,  out-the-window) 
information  about  local  airspace  and  terrain  features  (Martin-Emierson  &  Wickens,  1997).  Pilots 
have  attempted  to  develop  efficient  scanning  methods  to  cope  with  these  demands,  but  switching 
between  dom.ains  is  particularly  challenging,  because  it  requires  rapid  changes  in  !ine-of-sight 
(LOS)  and  optical  focus,  often  under  adverse  physical  (e.g.,  high-g)  conditions.  Such  switching  is 
also  cognitively  demanding,  since  pilots  must  pick-up  and  synthesize  information  across  domains, 
thus  creating  a  high  potential  for  er.mr  (Weinstein  &  Wickens,  1992). 

Visual  Target  Search.  The  need  for  rapid  target  acquisition  is  greatest  in  the  context  of 
threat  engagements,  where  pilots  must  locate,  identify,  track,  and  designate  potential  air  and 
ground  targets,  while  flying  complex  attack  maneuvers.  Prior  to  the  use  of  radar  and  other 
sensor-assisted  target  cueing,  pilots  relied  on  unaided  sight  in  detecting  and  attacking  the  enemy. 
The  deve'opmient  of  radar  provided  assistance  in  finding  targets  (Koonce,  1999).  However,  it  did 
not  eliminate  the  need  for  visual  search,  and  it  added  a  new  problem:  In  order  to  locate  targets 
and  also  maintain  system  status,  radar  combined  the  continued  need  for  out-the-window  (OTW) 
scans  {head-up  viewing)  with  the  need  to  look  down  at  displays  mounted  below  the  aircraft 
windshield  {head-down  viewing).  For  example,  during  the  dynamic  attack  phase  of  air-to-air 
combat,  pilots  must  attend  to  far-domain  events  (visually  acquiring  and  tracking  target  location) 
and  near-domain  data  sources  (e.g.,  monitoring  radar;  Barnes,  1 989),  an  arrangement  which,  as 
indicated  above,  is  far  from  ideal. 

Visual  Display  Approaches  to  Improving  Information  Presentation 

Head-Up  Display  (HUD).  Given  the  value  of  air  superiority  and  pilot  survival,  there  is 
great  impetus  to  improve  information  presentation  in  the  tactical  cockpit.  One  successful 
approach  has  been  to  design  interfaces  that  combine  the  presentation  of  information  from  near 
and  far  domains  and  thus  reduce  the  need  to  scan  the  domains  separately.  The  primary 
technology  to  serve  this  function  is  the  head-up  display  (HUD),  a  transparent  display  located 
between  the  pilot  and  the  aircraft  windshield  upon  which  navigational  and  tactical  information 
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(e.g.,  airspeed,  altitude,  target  location)  can  be  projected.  HUDs  are  generally  mounted  at  eye- 
height,  and  imagery  is  focused  at  optical  infinity  (Adam  1994).  Looking  through  the  HUD  allows 
pilots  to  view  the  outside  wor'd  and  sample  displayed  information  without  changing  their  line-of- 
sight.  Although  originally  developed  to  serve  as  aiming  reticles  or  weapo.ns  sights,  HUDs  now 
provide  pilots  with  additional  information,  such  as  navigational  data  and  vectors  to  fly  to  targets 
(Raising,  Liggett,  &  Munns,  1999).  During  target  search  tasks,  a  symbol  portrayed  on  the  HUD 
can  indicate  the  position  of  a  target  in  airspace  (Geiselmian  &  Osgood,  1995;  Geiselman  &  Tsou, 
1996),  thereby  eliminating  the  need  to  re-map  information  from  a  head-down  display  to  the  out- 
the-window  scene  (Barnes.  1989).  A  display  such  as  this,  which  directly  maps  near-domain 
symbology  onto  far-domain  external  objects,  is  termed  a  conformal  display. 

The  performance  benefits  of  the  HUD  have  been  demonstrated  in  several  different  flight 
tasks  in  termis  of  decreased  reaction  timie,  error  rate,  and  workload  (Fadden,  Ververs,  &  Wickens, 
1998;  Martin-Emerson  &  Wickens.  1997;  Weintraub  &  Ensing,  1992).  Moreover,  tactical  HUDs 
have  become  standard  in  most  modern  fighter/attack  aircraft,  including  the  United  States  Air 
Force's  F-15,  F-16,  and  F-1 17  (stealth  fighter).  However,  they  are  by  no  means  a  panacea  to 
problems  of  information  acquisition.  Like  head-down  displays,  HUDs  can  present  information  to  a 
pilot  only  when  within  his  or  her  field-of-view  (FOV).  While  a  pilot's  instantaneous  FOV  is 
approximately  +/- 100°  horizontal  by  +/-  60°  vertical  (Boff  &  Lincoln.  1988;  Velger  1998),  technical 
constraints  have  limited  the  HUD  FOV  to  under  40°  horizontal  and  vertical  (Burley  &  LaRussa, 
1990;  Velger,  1998;  Weintraub  &  Ensing,  1992),  a  relatively  small  portion  of  the  total  visible 
airspace.  For  example,  lateral  visibility  in  the  F-1 5  extends  approximately  1 50°  in  either  direction 
(Burley  &  LaRussa,  1990).  Accordingly,  pilots  are  often  unable  to  acquire  near-domain 
information  when  looking  out-the-window  (OTW).  Barnes  (1989)  has  estimated  that  during  the 
attack  phase  of  threat  engagements,  pilots  are  looking  out-the-window  and  away  from  the  HUD 
about  three  quarters  of  the  time. 

A  related  constraint  is  the  fixed  field-of-regard  (FOR)  with  respect  to  the  far  domain. 

Since  the  HUD  is  fixed  to  the  aircraft's  nose  {boresighf),  its  limited  field-of-view  can  be  modified 
only  by  moving  the  aircraft.  This  limitation  is  especially  serious  in  air  combat  when  the  HUD  is 
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used  as  a  target  designator  and  weapons  sight.  In  these  situations,  it  is  impossible  to  designate 
off-boresight  targets  and  it  is  often  necessary  to  deviate  from  optimal  attack  maneuvers  or  flight 
paths  in  order  to  move  the  aircraft's  boresight  into  appropriate  position  (Barnes,  1989;  Velger, 
1998).  The  development  of  high  off-axis  boresight  weapon  systems  (HOBS)  could  allow  for  the 
designation  of  targets  as  great  as  90°  off-nose  (Beal  &  Sweetman,  1994;  Velger,  1998).  Taking 
advantage  of  HOBS  weapons  capabilities  requires  an  interface  that  overcomes  the  HDD's  limited 
field-of-regard. 

Helmet-Mounted  Display  (HMD).  The  helmet-mounted  display  (HMD)  can  be  considered 
the  next  significant  development  in  tactical  cockpit  interfaces  following  the  introduction  of  the 
head-up  display.  There  are  two  genera!  oategories  of  HMDs  (Melzer  &  Moffitt,  1997;  Velger, 
1998).  Opaque  HMDs,  which  present  a  virtual  visual  environment,  are  used  prim.arily  for 
simulation  and  training.  See-through  HMDs,  derived  from  HDDs,  allow  conformal  information  to 
be  portrayed  on  a  transparent  visor  and  are  used  in  operational  situations  such  as  weapons 
targeting.  Since  a  see-through  HMD  projects  information  on  the  helmet  visor,  displayed 
symbology  is  always  within  the  field-of-regard.  In  order  for  HMDs  to  serve  as  effective  target- 
cueing  and  designation  systems,  two  additional  components  are  added.  A  tracker  measures 
head  position  in  real  time  so  that  information  is  presented  with  respect  to  the  pilot's  line-of-sight, 
rather  than  a  fixed  position,  such  as  the  aircraft  boresight.  A  helmet-mounted  sight  (HMS)  allows 
pilots  to  use  their  head  position  as  a  control  by  coupling  a  reticle  with  the  symbology  on  the  HMD 
(Adam  1994;  Velger,  1998).  The  integration  of  these  components  enables  presentation  of  target 
location  information,  irrespective  of  field-of-regard.  An  example  of  HMD  symbology  is  illustrated  in 
Figure  1 ,  in  which  a  locator  line  referenced  to  the  pilot's  head  position,  indicates  the  direction  to  a 
target  beyond  the  display  field-of-view.  Consequently,  pilots  can  smoothly  locate  and  track  a 
target,  and  if  appropriate,  use  the  aiming  reticle  to  designate  a  weapon  off-boresight  without  re¬ 
positioning  the  aircraft  (Barnes  1989;  Geiselman  &  Tsou,  1996).  Although  first  developed  in  the 
1960s,  airborne  HMD  designs  were  not  implemented  successfully  until  the  1980s,  awaiting 
advances  in  both  optical  and  weapon  systems  technologies  (Melzer  &  Moffitt,  1997;  Velger, 

1 998).  A  motivating  factor  for  the  development  of  these  technologies  in  the  U.S  military  has  been 
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the  implementation  of  helmet-mounted  sighting  system.s  by  the  South  African,  Russian,  and 
Israeli  airforces  (Bea!  &  Sweetman,  1994;  McQuillan,  1999). 


Figure  1.  HMD  target  localization  and  designation  symbology  (adapted  from  Geiselman  &  Tsou, 
1996). 

For  tactical  situations,  HMD  systems  have  a  number  of  advantages  over  head-up 
displays.  Figure  2  illustrates  the  field-of-regard  limitations  for  HUDs  relative  to  HMDs  with  respect 
to  the  directional  range  of  sensor  systems  including  radar  and  infrared  (FLIR).  Note  that  the  HUD 
has  a  relatively  narrow  field-of-regard  that  does  not  cover  the  entire  directional  range  of  sensors. 
In  comparison,  the  HMD,  which  accounts  for  head  movement,  enables  information  display  that 
exceeds  the  total  directional  range  for  sensors  (Merryman,  1994;  Velger,  1998). 

Although  much  of  the  research  on  operational  HMD  efficacy  remains  classified  (E.E. 
Geiselman,  Air  Force  Research  Laboratory,  personal  communication,  March  15, 1999),  several 
studies  are  available  In  the  literature  which  indicate  performance  benefits  for  HMDs.  Arbak, 

King,  Jauer,  and  Adam  (1988)  found  that  pilots  flying  simulated  air-to-air  combat  missions 
achieved  a  greater  number  of  kills  and  a  greater  exchange  rate  (ratio  of  kills  to  losses)  when 
using  HMDs  in  conjunction  with  HUDs  than  when  using  tactical  HUDs  alone.  Moreover,  the  pilots 
reported  a  number  of  advantages  to  HMD  use,  including  reduced  time-to-shoot  by  steering  their 
heads  instead  of  the  aircraft.  In  a  similar  study,  Olson,  Arbak,  and  Jauer  (1991)  reported  a  21% 
increase  in  exchange  rate  when  HMDs  supplemented  the  HUD. 
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Figure  2.  Field-of-regard  ranges  for  HUD,  sensors,  HMD,  and  potential  threats  (adapted  from 
Velger,  1998). 


HMDs  have  also  been  shown  more  effective  than  HUDs  in  air-to-ground  combat 
scenarios.  Since  ground  targets  can  be  masked  by  the  terrain,  they  are  typically  more  difficult  to 
locate.  Osgood,  Wells,  and  Meador  (1995)  compared  performance  with  and  without  HMD-based 
target  cueing  during  simulated  low-altitude  ground  attack  missions.  Compared  with  the  HUD 
alone,  use  of  the  HMD  resulted  in  releasing  weapons  more  rapidly  and  at  a  greater  distance  from 
targets.  Pilots  were  also  able  to  designate  targets  farther  off  boresight,  thereby  taking  fuller 
advantage  of  HOBS  weapons  capabilities.  Again,  the  participating  pilots  reported  HMD  speed 
advantages  for  target  designation. 

Helmet-mounted  displays  improve  the  linkages  between  pilot,  sensor,  and  weapons 
systems.  Freeing  the  pilot  from  a  fixed  boresight  field-of-regard  improves  both  target  acquisition 
and  kill  rate.  However,  HMDs  are  limited  in  their  utility.  Like  HUDs,  helmet-mounted  displays  have 
obstacles  that  limit  the  area  onto  which  information  may  be  projected.  For  a  fixed field-of-regard 
(i.e.,  with  the  head  stable),  see-through  HMDs  have  approximately  the  same  display  field  as 
HUDs  (Davis,  1 997;  Wells  &  Haas,  1990).  This  constraint  is  partly  determined  by  current  HMD 
designs  that  require  that  the  helmet-mounted  projection  system  be  supported  by  the  pilot's  head 
and  neck,  resulting  in  a  tradeoff  between  display  FOV  and  helmet  size  and  weight.  Increasing  the 
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display  area  would  require  an  increase  in  helmet  size,  which  is  constrained  by  the  ejection 
envelope  (cockpit-space  requirements  for  safe  pilot  ejection).  Additionally,  increasing  helmet 
size/weight  could  reduce  scanning  performance  and  endurance  and  increase  the  probability  and 
severity  of  head/neck  injury  (Dudfield,  Hardiman,  &  Selcon,  1995;  Perry  &  Buhrmian,  1997;  Wells 
&  Venturino,  1990).  Moreover,  while  a  limated  display  area  allows  for  relatively  unobstructed  sight 
in  see-through  HMDs,  it  necessitates  head  movem.ents  for  repositioning  symbology  (sluing), 
which  are  less  efficie.nt  than  saccadic  eye  movements  (Davis,  1997).  However,  Davis  (1997)  has 
pointed  out  that  extreme  eye  movements  required  by  larger  FOV  displays  would  be 
uncomfortable,  and  recommends  limiting  them  to  +/- 1 5°  of  the  central  !ine-of-sight.  The  optimal 
tradeoff  between  display  area,  weight,  and  performance  remains  to  be  determ.ined.  Given  this 
uncertainty,  it  seems  worthwhile  to  determine  whether  HMD  efficacy  can  be  enhanced  by 
providing  i.nformation  via  other  sensory  modalities. 

Auditory  Cueing 

Spatialized  Sound.  Since  the  perception  of  a  sound  source  is  limited  neither  by  field-of- 
regard  nor  line-of-sight,  several  researchers  have  proposed  that  auditory  cueing  be  used  to 
improve  visual  target  search  performance  (McKinley,  Ericson,  &  D'Angelo,  1994;  Perrott, 
Cisneros,  McKinley,  &  D'Angelo,  1996;  Rudmann  &  Strybel,  1999).  Perrott,  Saberi,  Brown,  and 
Strybel  (1990)  suggested  that  vision  and  audition  can  function  synergistically  in  search  tasks  by 
enabling  auditory  localization  to  direct  the  visual  system  toward  objects  in  space.  They  found  that 
presentation  of  a  spatially  coincident  free-f/e/d  sound  (originating  from  an  actual  source)  with  a 
visually  presented  target  resulted  in  significantly  reduced  search  times.  Gains  in  search 
efficiency  were  greatest  when  targets  were  presented  outside  participants’  fields-of-view. 
However,  auditory  cueing  still  reduced  search  times  within  the  central  1 0°  of  the  FOV  by  an 
average  of  175msec.  The  authors  suggested  that  aurally  conveyed  information  is  used  to  initiate 
and  modify  appropriate  head  movements. 

Perrott,  Sadralodabai,  Saberi,  and  Strybel  (1991)  more  closely  explored  the  effectiveness 
of  auditory  cueing  in  the  central  FOV  (<  15°).  The  effectiveness  of  the  sound  cue  was  again  a 
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positive  function  of  angular  distance  from  !ine-of-sight.  There  was  a  significant  benefit  in  search 
times  for  the  most  dista!  targets  of  about  lOOmssec,  but  no  benefit  for  targets  within  the  central 
line-of-sight  (<  These  studies  show  that  the  benefits  of  auditory  cueing  in  visual  search  tasks 
are  maximal  when  targets  are  beyond  an  observer's  centra!  field-of-view. 

Rudmann  and  Strybel  (1999)  proposed  that  the  utility  of  auditory  cueing  lies  in  its 
conveying  general  rather  than  precise  information  about  target  location.  To  explore  this 
possibility,  they  utilized  a  method  developed  by  Williams  (1973)  to  analyze  visual  search 
performance  in  two  phases,  localization  and  identification.  Localization  occurs  in  the  first  stage  of 
the  search  process,  where  a  target  is  brought  into  the  field-of-view.  The  precise  target  position 
within  the  field-of-view  is  determined  during  the  succeeding  identification  stage.  Rudmann  and 
Strybel  found  that  sound  helped  in  both  stages  of  the  task,  with  the  greater  benefit  of  auditory 
cueing  accruing  to  target  localization. 

Virtual  Auditory  Displays,  implementing  spatialized  sound  in  cockpits  requires  the  use  of 
synthetic  audio  generators  to  spatially  render  target  positions  and  deliver  them  to  binaural  cockpit 
headsets  as  positional  audio  cues.  The  use  of  digital  filtering  systems  allows  for  the  rendering  of 
sound  sources  at  any  positions  in  space  (Shinn-Cunningham,  Lehnert,  Kramer,  Wenzel,  & 
Durlach,  1997;  Wenzel,  Arruda,  Kistler,  &  Wightman,  1993).  These  filters  are  based  on  the 
measurement  of  head-related  transfer  functions  (HRTF):  that  is,  intensity  and  phase 
transformations  of  sound  resulting  from  reflection/  attenuation  by  the  listener's  body,  head,  and 
pinnae  (Wightman  &  Kistler,  1989  a,  b).  The  transfer  functions  may  be  individualized  (derived 
from  measurements  from  an  individual)  or  generic  (derived  from  a  representative  model).  In 
general,  since  only  a  limited  number  of  positional  sound  samples  are  taken  to  compute  HRTFs, 
mathematical  interpolation  is  required  to  render  sounds  between  sampled  positions,  contributing 
to  a  drop  in  resolution.  As  with  HMD  symbology,  spatialized  auditory  displays  can  be  coupled 
with  head-tracking  systems  to  allow  information  to  be  presented  with  respect  to  changing  head 
position  and  orientation. 

Although  virtual  audio  displays  do  not  provide  the  same  resolution  as  free-field  sound 
sources,  they  can  nevertheless  improve  performance  in  localization  tasks  (Bolia,  D’Angelo,  & 
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Mckinley,  in  press;  Perrott,  Cisneros,  McKinley,  &  D'Angelo,  1996).  Begault  (1993)  evaluated  the 
utility  of  three-dimensional  auditory  cueing'  (using  generic  HRTFs)  in  locating  moving  visual 
targets.  Flight  crews  flew  a  simulated  collision-avoidance  mission  in  which  they  were  required  to 
search  out-the-window  for  air  traffic.  Pilots  using  localized  sound  acquired  aircraft  over  two 
seconds  more  quickly  than  with  non-localized  auditory  warnings.  Similarly,  Bronkhorst,  Veltman, 
and  Breda  (1996)  found  that  when  presented  in  a  flight  simulator,  virtual  auditory  cues  were  of 
significant  value  in  helping  pilots  locate  and  track  target  aircraft,  especially  when  they  were 
presented  in  combination  with  visual  displays. 

Another  context  in  which  auditory  cueing  has  proven  effective  is  in  the  acquisition  of 
targets  located  within  highly  distracting  environments  (Perrott,  Sadralodabai,  Saberi,  &  Strybel, 

1 991 ).  Distracters  can  consist  of  a  variety  of  stimuli  ranging  from  elements  designed  to  look  like 
targets,  to  the  background  in  which  targets  are  embedded,  such  as  camouflaging  terrain.  Bolia, 
D’Angelo,  &  McKinley  (in  press)  evaluated  reaction  times  for  target  detection  in  varying  distracter 
ratio  environments  (ratio  of  number  of  distracters  to  a  target),  under  non-cueing,  tree-field,  and 
virtual  audio  conditions.  The  authors  demonstrated  that  virtual  auditory  cueing  not  only  enhanced 
acquisition  speed  to  near  that  of  free-field  sound,  but  also  significantly  minimized  the  negative 
effect  of  high-distracter  ratios.  Accordingly,  virtual  audio  may  be  particularly  useful  in  assisting 
the  localization  of  targets  under  non-optimal  visual  conditions,  such  as  ground  target  detection. 

Adding  sound  to  the  HMD  system  is  a  natural  extension  of  designing  displays  in  terms  of 
a  perceptual  system  rather  than  independent  senses  (cf.  Gibson,  1966).  Melzer  and  Moffitt 
(1997)  point  out  that  head-down  displays  essentially  nominate  vision  as  the  effective  perceptual 
system,  without  accounting  for  head-motion  nor  the  other  sense  modalities.  Helmet-mounted 
displays  can  be  referenced  to  head-position,  thereby  leveraging  the  functionality  of  vision. 
Accordingly,  it  makes  sense  to  include  audition,  which  can  influence  head  and  eye  movements 
(Perrott  et  al.,  1991;  Rudmann  &  Strybel,  1999),  as  a  part  of  the  interface  design.  This  is 
especially  relevant  for  helmet-mounted  sights  because  head  position,  in  conjunction  with  aircraft 

'  Begault  utilized  exaggerated  audio  cues  to  accentuate  localization  direction  in  the  horizontal  plane  by 
mapping  sound  sources  to  three  times  the  angular  distance  as  the  corresponding  visual  position.  For 
example,  a  target  that  was  1 5  degrees  off  axis  would  sound  as  if  it  were  45  degrees  off. 


9 


position,  controls  target  designation  systems. 

Recent  research  has  addressed  the  use  of  virtual  spatiaiized  audio  in  conjunction  with 
limited  field-of-view  HMDs  for  visual  target  search.  This  work  employed  opaque  rather  than  see- 
through  HMDs  to  present  a  low-fidelity,  vi.Tua!  visual  environment  in  which  participants  could 
search  for  targets  without  conformal  symbology.  Since  such  HMDs  have  highly  constrained  visual 
fields,  they  lend  themselves  readily  to  the  inclusion  of  other  modalities  (Davis,  1 997).  In  the 
absence  of  supplementary  visual  cueing  (e.g.,  locator  lines  -  see  Figure  1),  the  presence  of  virtual 
auditory  cueing  led  to  increased  detection  rates  and  decreased  detection  times,  and  to  reductions 
in  reported  workload  (Cunningham,  Nelson,  Hettinger,  Russell,  &  Haas,  1995;  Nelson,  Hettinger, 
Cunningham,  Brickman,  Haas,  &  McKinley,  1998).  in  addition  to  these  performance  and  workload 
benefits,  virtual  auditory  cueing  also  improved  the  efficiency  of  search  patterns,  as  revealed 
through  analyses  of  participants’  head  motion  (Cunningham.,  et  ai.  1995;  Nelson,  et  al.  1998). 

While  the  results  of  research  combining  virtual  auditory  with  HMD  technology  are 
impressive  and  theoretically  sound,  it  is  important  to  emphasize  that  they  did  not  provide 
supplementary  visual  cueing.  See-through  HMDs,  which  provide  conformal  symbology,  are  more 
common  in  operational  settings,  and  it  is  not  clear  whether  the  beneficial  effects  of  added  auditory 
cueing  would  also  accrue  to  these  types  of  HMDs.  More  specifically,  users  can  rely  on  the 
conformal  symbology  of  see-through  HMDs  (along  with  rich  views  of  the  outside  world)  to  locate 
targets.  Additionally,  Flanagan,  McAnally,  Martin,  Meehan,  and  Oldfield  (1998)  have 
demonstrated  that  when  visual  target  cueing  is  available  in  an  opaque  HMD,  auditory  cueing  may 
be  of  no  additional  benefit  in  aiding  a  visual  search  task.  Accordingly,  one  goal  for  the  present 
study  was  to  assess  the  effects  on  performance  and  perceived  workload  of  virtual  auditory  cueing 
on  target  acquisition  with  a  see-through  visual  display  and  sight. 

Interface  Management 

Cockpit  Clutter.  Helmet-mounted  displays  and  virtual  auditory  cueing  offer  performance 
and  workload  advantages  to  pilots;  however,  integrating  these  technologies  into  the  cockpit  is  not 
a  simple  matter.  It  requires  consideration  of  their  interactions  with  existing  interfaces  and  with 
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one  another.  Single-seat  aircraft  cockpits,  such  as  that  in  the  Air  Force's  premier  lightweight 
fighter,  the  F-1 6C  Falcon,  have  relatively  limited  space  for  information  display.  Moreover,  there  is 
already  a  high  display  load  in  cockpits,  and  too  many  displays  may  contribute  to  information 
overload,  increasing  the  probability  of  pilot  error  (Reising,  Liggett,  &  Munns,  1999;  Weinstein  & 
Wickens  1 992).  The  unlimited  field-of  regard  of  HMDs,  and  the  omnidirectonal  character  of 
auditory  displays  might  exacerbate  that  clutter. 

Adaptive  Interfaces.  An  em.erging  approach  to  counte.'acting  clutter  is  the  use  of 
adaptive  interfaces,  in  which  automation  is  emiployed  to  control  the  delivery  of  information  to 
pilots  so  that  they  receive  the  right  information  in  the  right  format  at  the  right  time,  and  are  not 
otherwise  exposed  to  that  information  (Hettinger,  Cress,  Brickman,  &  Haas,  1996;  Hollnagel, 
1988).  The  utility  of  such  a  situation-dependent  display  is  illustrated  in  a  study  by  Brickm.an, 
Hettinger,  Roe,  Lu,  Repperger,  and  Haas  (1996),  who  used  an  adaptive  force-reflecting  aircraft 
control  stick  to  aid  in  landings  during  calm  and  turbulent  flight  conditions.  The  control  stick 
operated  normally  when  the  aircraft  was  correctly  aligned  with  the  runway.  However,  whenever 
the  aircraft  deviated  laterally  from  a  pre-determined  flight  path,  the  stick  provided  increased 
resistance  against  control  inputs  that  would  further  increase  the  deviation.  The  control  stick 
offered  no  benefit  over  a  standard  stick  under  calm  flight  conditions,  but  under  turbulent 
conditions  aircraft  landings  were  more  accurate  with  the  haptically-augmented  control  stick  than 
with  its  standard  counterpart.  In  addition  to  landing  functions,  other  studies  have  shown  that 
providing  necessary  information  only  during  critical  situations  can  also  be  of  effective  value  to 
pilots  during  navigational  tasks  (Bennett,  Cress,  Hettinger,  Stautberg,  &  Haas,  in  press;  Moroney, 
1999).  Accordingly,  a  second  goal  for  the  present  investigation  was  to  evaluate  the  role  of 
adaptive  interfaces  as  a  means  of  integrating  visual  and  auditory  displays  for  target  designation. 

In  brief,  it  was  expected  that  when  compared  to  unimodal  and  non-cued  conditions,  multimodal 
audio-visual  interfaces  would  lead  to  higher  target  detection  rates  and  faster  search  times, 
particularly  for  targets  that  were  initially  outside  of  the  display  symbology  field-  of-view.  It  was  also 
anticipated  that  performance  with  multimodal  adaptive  interfaces,  which  reduce  cockpit  clutter, 
would  be  superior  to  that  associated  with  fixed-format  multimodal  interfaces. 
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CHAPTER  2 


Method{tc  \!1  "METHODS  } 

Participants{tc  \12  "Participants} 

Twelve  pilots  (11  males  and  1  female)  serving  at  Wright-Patterson  Air  Force  Base 
participated  in  the  stjdy.  They  ranged  in  age  from  32  to  51  years,  with  a  mean  of  40  years.  All 
had  norm.al  or  corrected-to-normal  vision,  based  upon  Snellen  acuity,  and  normal  auditory 
functioning:  threshold  at  500Hz  to  4000Hz:  25dB  SPL  or  better  (ANSI,  1969).  Each  pilot  had  at 
least  1500  hours  of  flight  experience  in  military  aircraft,  such  as  training  (T-37:  T-38),  fighter  (F-4; 
F-15;  F-16),  and  transport  (C-130;  KC-1 35)  jets,  with  a  mean  of  2652  hours  logged. 


Experimental  Design{tc\l3  "Experimental  Design} 

As  shown  in  Table  1 ,  seven  target-cueing  conditions  were  employed  in  this  experiment. 
They  included  a  non-cueing  control  and  six  cueing  interfaces.  The  cueing  interfaces  featured 
unimodal  auditory,  unimodal  visual,  and  multimodal  (auditory  and  visual)  displays  presented  in 
fixed  or  adaptive  configurations.  All  pilots  participated  in  all  conditions. 


Table  1 

Cueing  Conditions. 


Target  Cueing  Interfaces _ 

Non-Cueing  [Non]{tc  \I3  “Non-Cueing} 

Fixed  Format:  Auditory  [Aud] 

Fixed  Format:  Visual  [Vis] 

Fixed  Format:  Auditory  plus  Visual  [A+Vj 

Adaptive  Format:  Adaptive  Auditory  plus  Adaptive  Visual  [Adap(A+V)] 
Adaptive  Format:  Adaptive  Auditory  plus  Visual  [Adap(A)+V] 

Adaptive  Format:  Adaptive  Visual  plus  Auditory  [Adap(V)+A] 
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Apparatus  and  Procedures{tc  \13  “Apparatus  and  Procedures} 

Laboratory  Facilities.  The  study  was  conducted  at  the  Air  Force  Research  Laboratory's 
Synthesized  Immersion  Research  Environment  (SIRE)  at  Wright-Patterson  Air  Force  Base.  As 
illustrated  in  Figure  3,  the  SIRE  facility  houses  a  simulated  F-16C  cockpit  situated  in  the  center  of 
a  20'  radius  dome  that  includes  a  high-resolution,  large  field-of-view  (70°  vertical  by  150° 
horizontal)  interactive  visual  display.  For  experimental  testing,  the  cockpit  was  raised  7.5'  to  place 
pilots’  eyeheight  at  the  approximate  vertical  midpoint  of  the  dome.  The  fixed-base  cockpit 
contained  head-up  and  head-down  visual  displays  and  F-1 6  throttle  and  sidestick  aircraft 
controls.  Out-the  window  (OTW)  imagery  consisted  of  desert-like  terrain  with  rolling  hills 
(southwest  New  Mexico,  see  Figures  4a, b).  It  was  generated  by  a  Silicon  Graphics  Onyx  Reality 
Engine  2  computer  and  presented  by  a  set  of  six  Barco  cathode  ray  tube  projectors,  each 
providing  1280  horizontal  x  1024  vertical  pixel  resolution.  The  beams  from  the  six  projectors  were 
calibrated  to  provide  a  single,  coherent  image  on  the  dome  display.  The  Reality  Engine  also 
generated  air  and  ground  targets,  as  well  as  the  visual  target  cueing  and  designation  symbology. 
All  additional  visual  (cockpit  displays)  and  auditory  (target  cueing  and  performance  feedback) 
stimuli  were  generated  by  Intel  processor-driven  personal  computers,  which  controlled  trial 
sequences  and  collected  data. 

Flight  Task.  Pilots  were  instructed  to  maintain  an  airspeed  of  500  knots  while  flying  at  or 
below  300'  AGL  (above  ground  level).  The  flight  task  also  required  pilots  to  minimize  lateral 
deviation  from  a  waypoint-guided  path.  There  were  three  waypoints,  or  turning  points,  during  a 
flight,  which  were  spaced  approximately  19.5  miles  apart  (see  Figure  5).  Pilots  determined 
deviation  from  the  flight  path  through  a  Horizontal-Situation-Display  (HSD),  which  was  located  in 
the  left  side  of  the  cockpit.  As  illustrated  in  Figure  6,  this  head-down  display  portrayed  a  top-down 
view  with  the  pilot’s  aircraft  at  the  center.  The  circles  indicated  waypoints  and  the  lines 
connecting  the  waypoints  depicted  the  flight  path.  Additionally,  a  head-up  display  (HUD),  having  a 
30°  field  visual  field,  provided  avionics  information,  including  aircraft  heading,  airspeed,  and  AGL 
altitude,  as  shown  in  Figure  7. 
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Figure  4a.  Detail  of  dome-projected  out-the  window  graphics  and  HMD  symbology.  This  image, 
which  is  approximately  30  x  30  degrees  visual  angle,  shows  a  ground  target  approximately  3000’ 
away  at  terrain  level. 


Figure  4b.  Detail  of  dome-projected  out-the  window  graphics  and  HMD  symbology.  This  image, 
which  is  approximately  30  x  30  degrees  visual  angle,  shows  an  air  target  approximately  500’ 
away  and  300’  above  the  terrain. 
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Figure  5.  Typical  flight  path  superimposed  over  section  of  New  Mexioo  terrain  map.  Starting 
position  and  navigational  waypoints  are  indicated. 


Figure  6.  The  horizontal  situation  display  (HSD)  presented  an  aircraft-centric  view  of  the  flight 
path  and  waypoints,  allowing  a  pilot  to  determine  and  correct  lateral  deviation  from  the  path. 


16 


Airspeed  - 


-  650 

12- 

-600 

11- 

-550  ^ 

10- 

-500 

35  0  1  2 

9  - 

till 

.A 

85D02 

Ciurent  Heading 


500 


'Altitude 


Radar  Altimeter 


Figure  7.  The  head-up  display  (HUD)  provided  airspeed,  above-ground-level  altitude,  and  course 
deviation  information. 


Target  Designation  Task.  While  maintaining  the  directed  flight  path,  airspeed,  and  AGL 
altitude,  pilots  visually  searched  out-the-window  for  ground  (SCUD  launcher  vehicles.  Figure  4a) 
and  air  (MH-53  helicopters,  Figure  4b)  targets.  Each  trial  contained  eight  ground  and  eight  air 
targets  which  were  distributed  randomly  along  the  flight  path,  with  the  following  four  constraints. 
(1)Targets  ranged  laterally  between  2000'  and  4000'  from  the  flight  path;  (2)  each  target  was  a 
minimum  of  15000'  from  subsequent  targets,  so  that  pilots  would  only  have  to  locate  one  at  a 
time;  (3)  ground  targets  were  positioned  at  ground  level,  air  targets  at  300'  AGL;  (4)  targets  did 
not  appear  until  they  were  within  12000'  of  the  pilot’s  aircraft. 

Pilots  were  instructed  to  visually  locate  and  designate  targets  as  quickly  as  possible. 
Designation  was  accomplished  by  moving  a  head-coupled  aiming  reticle  (cross  hairs;  Figures 
4a,b)  over  a  target  and  depressing  the  appropriate  control-stick  button:  weapons  release  for 
ground  targets,  trigger  for  air  targets.  The  reticle  was  projected  on  the  dome  outside  the  cockpit, 
but  was  coupled  with  pilot  head  position,  so  that  it  simulated  a  helmet-mounted  sight.  Head 
position  was  measured  by  an  Ascension  Technology  Flock  of  Birds  electro-magnetic  tracker,  with 
the  receiver  mounted  on  a  flight  helmet.  The  helmet  also  housed  a  set  of  Sennheiser  HD  250  II 
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binaural  headphones  for  the  delivery  of  auditory  stimuli.  A  correct  designation  (hit)  required  pilots 
to  depress  the  proper  control-stick  button  while  the  cursor  was  within  3°  of  the  target.  Feedback 
for  hits  consisted  of  a  1 .25sec  change  in  cursor  color  (green  to  red),  paired  with  a  simultaneous 
lOOOhz  non-spatialized  tone,  presented  at  an  intensity  of  70dBA.  Targets  could  not  be  designated 
more  than  once. 

Target  Location  Cueing  Interfaces.  Under  all  but  the  Non-Cueing  condition,  visual  and/or 
auditory  cues  were  provided  to  aid  in  locating  targets.  In  situations  where  visual  cueing  was 
provided,  the  reticle  was  augmented  by  target  location  and  range  symbology  which,  like  the 
reticle,  was  projected  on  the  dome  and  slaved  to  the  pilots’  head  position  to  simulate  a  see- 
through  HMD.  Target  location  was  displayed  by  an  arrow  that  originated  from  the  center  of  the 
reticle.  As  shown  in  Figures  4a,b,  the  length  of  the  arrow  represented  the  angular  distance 
(distance  in  degrees  from  head  position)  to  the  target.  It  was  scaled  so  that  its  linear  extent 
would  not  exceed  15°  visual  angle.  The  direction  of  the  arrow  indicated  to  pilots  which  way  to 
look.  Range  to  target  was  portrayed  by  a  circle  (30°  diameter)  surrounding  the  reticle.  A  full 
circle  would  be  displayed  at  the  onset  of  cueing  (range  =  12000').  The  displayed  circumference 
of  the  circle  would  decrease  with  decreasing  target  range  (e.g.,  a  half  circle  at  6000',  a  quarter 
circle  at  3000  ). 

Where  auditory  cueing  was  provided,  white  noise  pulses  (250msec  at  70dBA)  were 
employed  to  portray  target  range  and  position.  Target  range  was  indicated  by  inter-pulse 
intervals,  which  decreased  from  a  maximum  of  1000msec  at  12000',  to  a  minimum  of  560msec  at 
1600'.  Target  position  was  indicated  by  digitally  spatializing  the  sound  in  azimuth  and  elevation 
with  respect  to  the  instantaneous  head  position  by  means  of  a  Convolvotron  audio  rendering 
system.  Spatialization  was  based  on  generic  head  related  transfer  functions  (HRTF)®. 

In  the  case  of  the  fixed  displays  (Auditory  [AUD],  Visual  [VIS],  and  Auditory  plus  Visual 
[A-^V]),  auditory  and/or  visual  cues  appeared  whenever  a  target  was  present.  In  the  case  of 
adaptive  displays,  interface  switching  was  determined  by  the  presence  of  the  target  within  the 

^  Localization  transformations  were  performed  by  interpolating  position  based  on  a  data  set  containing  72 
fixed-distance  filter  pairs  (left  and  right  ears).  These  pairs  were  sampled  at  15-degree  increments  from  six 
elevations  on  a  sphere. 
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helmet-mounted  display’s  field-of-view  (FOV).  As  suggested  by  Davis  (1997),  FOV  was  set  at  a 
30°  degree  circular  aperture,  thereby  limiting  excessive  eye  movement  and  reflecting  the  limited 
viewing  area  of  many  helmet-mounted-display  (HMD)  systems  (Adam,  1994;  Melzer,  1998;  Wells 
&  Venturino,  1990).  Thus,  in  the  Adaptive  Auditory  plus  Adaptive  Visual  [Adap(A+V)]condition,  a 
target  initially  beyond  the  display  FOV  (+/- 15°  in  any  direction)  would  be  cued  aurally.  When  the 
pilot  moved  his  or  her  head  to  encompass  the  target  in  the  display  area,  the  auditory  cueing  was 
replaced  by  the  HMD-based  visual  cueing  symbology.  In  the  Adaptive  Auditory  plus  Visual 
[Adap(A)+V]  condition,  auditory  cueing  was  provided  while  a  target  was  present  outside  the  FOV, 
while  visually  cueing  remained  regardless  of  FOV.  In  the  Adaptive  Visual  plus  Auditory 
[Adap(V)+A]  condition,  visual  cueing  was  only  presented  when  the  target  was  within  the  FOV, 
while  auditory  cueing  was  provided  whenever  a  target  was  present.  In  all  conditions  that  provided 
localization  cues,  auditory  and/or  visual  cueing  disappeared  whenever  a  correct  designation  was 
made,  or  if  the  target  was  bypassed. 

Experimental  protocol.  For  each  cueing  or  interface  condition,  pilots  completed  a  block  of 
five  consecutive  trials  in  which  they  flew  from  waypoint  to  waypoint,  searching  for  targets  while 
maintaining  the  appropriate  flight  parameters.  Crashing  into  terrain  or  flying  above  2000’  resulted 
in  a  failed  trial,  which  was  re-run  at  the  end  of  the  trial  block.  Each  block  lasted  approximately 
330sec.  Immediately  following  the  completion  of  each  experimental  condition,  pilots  assessed 
the  perceived  mental  workload  of  the  task  (the  information-processing  load  imposed  by  the  task, 
Eggemeier,  1988;  O’Donnell  &  Eggemeier,  1986)  via  the  NASA-Task  Load  Index  (TLX),  a  widely 
used  instrument  for  measuring  subjective  workload,  which  provides  a  reliable  index  of  overall 
workload  (test-retest  correlation  =.  83)  on  a  scale  of  0  to  100  (Hart  &  Staveland,  1988;  Hill, 
lavecchia,  Byers,  Zaklad,  &  Christ,  1992;  Nygren,  1991;  Procter  &  Van  Zandt,  1994). 

Pilots  were  afforded  a  5-minute  break  between  cueing  conditions.  The  order  in  which 
they  experienced  the  several  interface  conditions  as  they  progressed  through  the  experiment  was 
determined  at  random  for  each  individual.  The  first  two  conditions  to  be  experienced  were  tested 
on  the  first  day  of  a  pilot’s  participation  in  the  study,  with  the  remaining  conditions  tested  on  a 
subsequent  day.  The  two  testing  sessions  were  separated  by  no  more  than  two  days.  In  all 
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experimental  conditions,  data  regarding  search  performance  were  collected  in  terms  of  target 
designation  accuracy  and  time  (the  time  needed  to  correctly  designate  a  target  after  it  appeared). 
In  addition,  data  on  flight  performance  (deviation  from  flight  parameters)  and  head  motion  were 
recorded  five  times/sec  during  each  triai.  All  data  collection  was  managed  by  computer. 

Training.  Prior  to  reporting  for  the  experiment,  pilots  reviewed  detailed  written 
descriptions  of  the  flight  tasks  and  cueing  interfaces.  They  received  approximately  two  hours  of 
verbal  and  hands-on  training  on  the  first  day  of  participation.  Under  each  experimental  condition, 
training  trials  were  run  until  the  flight  path  was  fiown  to  its  compietion  with  a  minimum  of  25%  of 
the  targets  properly  designated.  All  pilots  completed  the  training  phase  successfully.  On  the  first 
day  of  participation,  pilots  were  given  a  10-minute  rest  between  the  end  of  training  and  the 
beginning  of  testing. 
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CHAPTER  3 


Results 


Visual  Search  Performance 

Target  Designation  Accuracy.  The  percentage  of  correct  target  designations  (hits)  across 
the  five  trial  blocks  was  determined  for  each  participant  for  each  cueing  condition.  Mean 
percentages  of  hits  for  targets  that  were  initially  outside  (+/-15“  from  the  reticle)  and  initially 
inside  of  the  display  fieid-of-view  (FOV)  are  plotted  for  the  seven  interface  conditions  in  Figure  8. 
Data  for  ground  and  air  targets  are  presented  in  the  upper  and  lower  panels,  respectively.  It  can 
be  seen  in  the  figure  that  with  the  exception  of  the  Non-Cueing  [Non]  ground  target  condition,  the 
hit  rate  was  greater  for  targets  initially  within  the  FOV  (M=  83%)  than  for  those  initially  outside  of 
the  FOV  (M  =  75%)  and  that  in  both  FOV  locations,  scores  for  the  Non-Cueing  [Non]  and  the 
Auditory  [Aud]  conditions  were  consistently  lower  than  those  for  the  other  conditions.  It  can  also 
be  seen  in  the  figure  that  the  overall  frequency  of  correct  target  designations  was  greater  for  air 
(M  =  89%)  than  for  ground  (M  =  69%)  targets. 

The  data  of  Figure  8  were  converted  to  arcsines  (Winer,  Brown,  &  Michels,  1991)  and 
tested  for  statistical  significance  by  means  of  a  2  (target)  x  2  (FOV)  x  7  (interface  condition) 
completely  repeated  measures  analysis  of  variance  (ANOVA)^  In  this  and  all  subsequent 
ANOVAs,  Box’s  epsilon  was  used  in  computing  degrees  of  freedom  to  correct  for  violations  of  the 
sphericity  assumption  as  determined  by  Mauchly’s  test  (Maxwell  &  Delaney,  1 990).  Significant 
main  effects  were  found  for  target  type,  F(1 ,11)  =  463.6,  g  <  .05,  to^  =  .26,  FOV,  F(1, 11)  =  99.89, 
E  <.05,  (0^  =  .08,  and  interface  condition,  F(3,  40)  =  61 .64,  g_<  .05,  =  .37,  and  there  was  a 

significant  target  type  x  interface  interaction  ,  F(3, 40)  =  17.20,  g  <  .05,  <x>^=  .05.  All  of  the 
remaining  sources  of  variance  in  the  analysis  were  not  significant,  g_>  .05  in  each  case. 


®  Complete  summaries  of  the  results  for  this  and  the  other  analyses  of  variance  in  this  study  can  be  found  in 
Appendix  A. 
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Air  Targets 


Figure  8.  Mean  percentages  of  ground  (upper  panel)  and  air  (lower  panel)  target  designations  for 
the  seven  interface  conditions.  Data  are  provided  for  targets  initially  outside  and  inside  the 
central  field-of-view  (FOV).  Error  bars  are  standard  errors. 
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The  Target  x  Interface  interaction  is  displayed  in  Figure  9.  Mean  percentages  of  correct 
air  and  ground  target  designations  are  shown  for  the  seven  interface  conditions.  For  each  type  of 
target,  separate  ANOVAs  revealed  that  there  were  statistically  significant  differences  among  the 
interfaces,  Fground(3,  40)  =  77.50,  b.<  .05,  co^  =  .84;  Fair  (3,  40)  =  1 8.00,  g  <  .05,  to^  =  .57. 
Supplementary  Newman-Keuls  tests  (alpha  =  .05  for  each  comparison  )  indicated  that  the  hit  rate 
for  ground  targets  in  the  Auditory  [Aud]  condition  was  significantly  greater  than  that  for  the  Non- 
Cueing  [Non]  condition,  and  the  means  for  each  of  these  conditions  were  significantly  lower  than 
those  for  each  of  the  five  remaining  interface  conditions.  The  scores  for  these  latter  conditions, 
however,  did  not  differ  significantly  from  each  other.  With  regard  to  air  targets,  the  Newman-Keuls 
tests  yielded  an  identical  outcome,  with  the  exception  that  the  Non-Cueing  and  Auditory 
conditions  did  not  differ  significantly  from  each  other. 


Figure  9.  Mean  percentages  of  ground  and  air  target  designations  (collapsed  over  field-of-view) 
for  the  seven  interface  conditions.  Error  bars  are  standard  errors. 


Designation  Time.  Designation  time  scores  for  each  participant  were  determined  by 
averaging  across  trial  blocks  within  each  cueing  condition.  Mean  designation  times  for  targets 
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initially  outside  and  initially  inside  of  the  FOV  are  plotted  for  the  Visual  cueing  and  the  four 
auditory  and  visual  cueing  conditions  in  Figure  10.  Data  for  ground  and  air  targets  are  presented 
in  the  upper  and  lower  panels,  respectively.  Scores  for  the  Non-Cueing  and  the  Auditory  cueing 
conditions  were  excluded  from  this  analysis  because  their  relatively  low  hit  rates  would  severely 
restrict  the  number  cases  in  which  correct  designation  times  were  based,  and  thereby  lower  the 
reliability  of  the  those  values.  The  most  conspicuous  feature  of  the  data  of  Figure  10  is  the 
considerable  difference  in  the  time  of  designation  for  ground  (M  =  5089msec)  in  comparison  to  air 
targets  (M  =  3696msec). 

The  designation  time  scores  were  tested  for  statistical  significance  by  means  of  a  2 
(target)  x  2  (FOV)  x  5  (interface)  completely  repeated  measures  ANOVA.  The  ANOVA  confirmed 
that  air  targets  were  designated  significantly  more  rapidly  than  ground  targets,  F(1 ,1 1)  =  969.25, 
Q_<  .05,  co^=  .73.  In  addition,  the  analysis  revealed  that  designation  times  were  significantly  faster 
for  targets  initially  outside  the  FOV  (  M  =  4301  msec)  than  for  those  initially  inside  the  FOV  (M  = 
4484msec),  F(1 ,11)  =  12.44,  p  <.05,  =  .01 ,  and  that  designation  times  differed  significantly 

among  the  interface  conditions.  Designation  times  in  the  Visual  [Vis]  cueing  condition  tended  to 
be  greater  than  those  in  all  of  the  conditions  featuring  audio-visual  cueing  (Mvis,  =  4597msec, 
MAdap(V)+A.  =  4490msec,  Maw,  =  4314msec,  MAdap(A)w,  =  4293msec  ,  MAdap(AH-v),  =  4269msec);  F(2, 
20)  =  5.21 ,  e  <  .05,  co^  =  .02.  In  addition  to  the  main  effects,  all  of  the  two-way  interactions  in  the 
analysis  were  statistically  significant:  Target  x  FOV,  F(1 ,11)  =  6.80,  p.<  .05,  (o^=  .02;  Target  x 
Interface  F(2,  20)  =  7.36,  e.<  .05,  co^  =  .73;  FOV  x  Interface,  F(2,  20)  =  4.36,  p  <  .05,  (0^=  .01 ,  as 
was  the  higher-order  interaction  between  target,  FOV,  and  interface,  F(2,20)  =  6.85,  p  <  .05,  (o^  = 
.01. 

The  nature  of  the  three-way  interaction  can  be  seen  in  Figure  10.  The  longer  time  needed 
for  target  designation  in  the  Visual  as  compared  to  all  of  the  audio-visual  cueing  conditions, 
approximately  825msec,  is  apparent  for  ground  targets  appearing  outside  the  field  of  view.  The 
consistent  disadvantage  for  the  Visual  cueing  condition  is  not  apparent  for  ground  targets  inside 
the  field  of  view  nor  for  air  targets  whether  inside  or  outside  the  field  of  view. 
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Figure  10.  Mean  times  to  designate  ground  (upper  panel)  and  air  (lower  panel)  targets  for  five  of 
the  interface  conditions.  Data  are  provided  for  targets  initially  outside  and  inside  the  central  field- 
of-view  (FOV).  Error  bars  are  standard  errors. 


Head  Motion 


An  analysis  of  head  motion  over  time  was  carried  out  within  each  interface  condition. 
Figures  1 1  a  through  1 1  g  portray  the  track  of  head  motion  in  azimuth  (angular  deviation  in  the 
horizontal  plane  from  the  aircraft  boresight)  and  elevation  (angular  deviation  in  the  vertical  plane 
from  the  aircraft  boresight )  for  several  subjects  during  a  representative  trial  for  each  of  the 
interface  conditions.  It  can  be  seen  in  Figure  11a  that  the  Non-Cueing  condition  resulted  in 
repetitive  head  motions  (indicated  by  the  density  of  the  track)  that  ranged  widely  in  azimuth 
(signified  by  the  spread  of  the  track),  suggesting  that  pilots  were  making  active  and  continuous 
visual  sweep  searches.  In  the  Auditory  cueing  condition,  Figure  11b,  there  is  a  drop  in  the  density 
and  range  of  head  motion.  Figures  for  the  subsequent  interface  conditions,  1 1  c-1 1  g,  reveal 
dramatic  declines  in  the  repetitiveness  (density  of  the  track)  and  lateral  deviation  of  the  pilots’ 
head  movements,  declines  which  appear  greater  than  in  the  auditory  alone  condition. 

Head  position  data  were  used  to  calculate  a  single  metric  of  head  motion,  angular  head 
displacement  -  the  total  angular  distance  the  head  traveled  throughout  an  experimental  trial’*. 

The  mean  angular  head  displacement  over  trials  is  provided  for  each  interface  condition  in  Figure 
12.  The  data  for  one  participant  were  dropped  for  the  purposes  of  this  analysis,  as  a  preliminary 
review  of  the  data  indicated  that  this  participant’s  head  motion  was  excessive  (>  2  S.D.  above  the 
group  mean). 

The  data  of  Figure  12  were  subjected  to  a  repeated  measures  ANOVA,  which  revealed  a 
significant  difference  among  the  interface  conditions,  F(2,20)  =  187.03,  p  <  .05,  o)^  =  .75. 
Supplementary  Newman-Keuls  tests  (alpha  =  .05  for  each  comparison)  indicated  that  the  degree 
of  angular  head  displacement  was  significantly  greater  in  the  Non-Cueing  [Non]  and  Auditory 
[Aud]  conditions  than  in  any  of  the  other  conditions  and  that  the  Non-Cueing  condition  was 
associated  with  greater  angular  head  displacement  than  the  Auditory  condition.  None  of  the 
comparisons  among  the  remaining  cueing  conditions  was  significant. 


'*  Angular  distance  was  determined  by  the  sum  of  the  differences  between  consecutive  head  positions  over  a 
trial,  which  were  computed  by  the  formula:  tan-1[(  (xi  -  xl-1)2  +  (yi  -  yl-1)2  +  (Zi  -  Zl-1)2  )]1/2.  Where 
{x,y,z)  is  a  point  on  a  sphere,  given  that  “i”  refers  to  the  initial  position  and  “I”  to  the  last  position  for  any 
consecutive  pair  of  head  positions  (Cunningham,  Nelson,  Hettinger,  Haas,  &  Russell,  1996). 
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Figures  11a-11g.  Head  motion  plots  in  azimuth  (Az.)  and  elevation  (El.)  for  a  representative  trial 
in  each  of  the  seven  interface  conditions.  Axes  are  scaled  to  the  dimensions  of  the  out-the- 
window  scene  (150°  horizontal  x  70°  vertical). 
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Figure  12.  Mean  angular  head  displacement  for  the  seven  interface  conditions.  Error  bars  are 
standard  errors. 


Flight  Performance 

Deviations  from  prescribed  airspeed,  flight  path,  and  altitude  (AGL)  were  used  to 
determine  the  degree  to  which  the  different  interface  conditions  affected  flight  performance. 

Airspeed  Error.  Root-mean-square  (RMS)  airspeed  error  (above  or  below  500  knots  or 
nautical  mph)  averaged  over  trials  is  presented  for  each  interface  condition  in  Figure  1 3.  A 
repeated  measures  ANOVA  of  the  airspeed  error  data  revealed  no  significant  differences  among 
the  interface  conditions  F(2,20)  =  3.00,  p  >  .05. 

Lateral  Error.  Lateral  error  was  measured  as  the  RMS  of  horizontal  deviations  in  feet 
from  the  flight  path  averaged  over  trials.  Mean  lateral  RMS  errors  for  the  seven  interface 
conditions  are  presented  in  Figure  14.  It  is  evident  in  the  figure  that  the  Auditory  [Aud]  condition 
was  associated  with  the  greatest  amount  of  error.  A  repeated  measures  ANOVA  of  the  lateral 
error  data  together  with  supplementary  Newman-Keuls  tests  (alpha  =  .05  for  each  comparison) 
confirmed  that  there  were  significant  differences  among  the  interfaces,  F(3,30)  =  4.32,  p  <  .05, 
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.17,  and  that  the  Auditory  interface  led  to  a  greater  amount  of  lateral  error  than  each  of  the 
other  conditions  except  the  Non-Cueing  condition.  No  other  comparison  among  the  interface 
conditions  was  statistically  significant. 

Above-Ground-Level  Error.  AGL  (above-ground-level)  error  was  measured  in  terms  of 
the  RMS  of  deviations  above  300  feet  AGL  averaged  over  trials.  Mean  AGL  errors  for  the  several 
interface  conditions  are  displayed  in  Figure  15.  A  repeated  measures  analysis  of  variance  of  the 
AGL  error  data  revealed  that  there  were  significant  differences  among  the  interfaces  F(4,40)  = 

3.1 1 ,  E  <  .05,  0)^  =  .09.  Supplementary  Newman-Keuls  tests  (alpha  =  .05  for  each  comparison), 
used  to  further  specify  the  nature  of  these  differences  indicated  that  the  Visual  [Vis]  interface 
condition  was  associated  with  a  significantly  smaller  degree  of  AGL  error  than  any  of  the  other 
conditions  except  the  Non-Cueing  [Non]  and  the  Adaptive  Auditory  plus  Adaptive  Visual 
[Adap(A+V)]  conditions.  All  other  comparisons  among  the  interface  conditions  were  not 
statistically  significant  (e  >  .05). 
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Figure  13.  Mean  RMS  deviations  from  prescribed  airspeed  (500  knots)  in  knots  for  the  seven 
interface  conditions.  Error  bars  are  standard  errors. 
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Subjective  Workload 

Mean  global  workload  ratings  on  the  NASA-TLX  are  presented  for  the  seven  interface 
conditions  in  Figure  16.  It  can  be  seen  in  the  figure  that  ratings  for  the  Non-Cueing  [Non]  and 
Auditory  [Aud]  conditions  fell  within  the  upper  half  of  the  workload  scale,  and  were  approximately 
50%  greater  than  those  for  the  remaining  interfaces  which  fell  within  the  lower  half  of  the  scale. 

A  repeated  measures  ANOVA  confirmed  that  there  were  statistically  significant  differences  in  the 
workload  associated  with  the  several  interfaces,  F(3,30)  =  12.80,  g  <  .05,  co^  =  .30. 
Supplementary  Newman-Keuls  tests  (alpha  =  .05  for  each  comparison)  indicated  that  the  overall 
workload  associated  with  both  the  Non-Cueing  and  Auditory  interfaces  was  greater  than  that  for 
each  of  the  other  interfaces.  The  workload  ratings  for  both  the  Non-Cueing  and  Auditory 
conditions  did  not  differ  significantly  from  each  other,  and  none  of  the  remaining  comparisons 
among  the  interface  conditions  reached  significance  (g  >  .05). 
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Figure  16.  Mean  global  workload  ratings  for  the  seven  interface  conditions.  Error  bars  are 
standard  errors. 
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CHAPTER  4 


Discussion 

In  order  to  meet  the  demands  of  tactical  missions,  it  is  necessary  for  pilots  to  rapidly 
acquire  and  act  on  information  from  sources  both  within  (near-domain)  and  beyond  (far-domain) 
the  cockpit.  Conformal  displays  such  as  HDDs  and  see-through  HMDs  were  designed  to  reduce 
the  demands  and  errors  associated  with  switching  line-of-sight  between  domains.  Although  these 
displays  enhance  visual  search  performance  for  both  ground  and  air  targets  (Arbak,  King,  Jauer, 
and  Adam,  1988;  Fadden,  Ververs,  &  Wickens,  1998;  Martin-Emerson  &Wickens,  1997;  Olson, 
Arbak,  and  Jauer,  1991;  Osgood,  Wells,  and  Meador,  1995;  Weintraub  &  Ensing,  1992),  their 
utility  is  constrained  by  their  limited  field-of-view.  The  purpose  of  the  present  study  was  to 
determine  the  efficacy  of  supplementing  a  simulated  HMD  with  spatialized  auditory  cues 
presented  in  unimodal,  mutlimodal,  or  adaptive  formats  during  a  flight-based  target  search  task. 
Audio-visual  cueing  has  been  shown  to  bolster  the  likelihood  of  target  detection  in  visual  search 
tasks  and  to  reduce  search  time  and  workload  (Bronkhorst,  Veltman,  &  Breda,  1996;  Nelson, 
Hettinger,  Cunningham,  Brickman,  Haas,  &  McKinley,  1998).  However,  these  findings  are  based 
upon  studies  that  did  not  provide  supplementary  visual  cueing  via  see-through  HMDs,  the  type  of 
HMD  most  often  used  in  tactical  aviation. 

Visual  Search  Performance 

Target  Designation  Accuracy.  In  order  to  destroy  targets,  pilots  must  locate  and 
designate  them  correctly.  Thus,  in  this  investigation  the  percentage  of  targets  correctly 
designated  was  of  principal  concern.  The  major  finding  with  this  measure  was  that  the  Non- 
Cueing  and  unimodal  Auditory  cueing  conditions  resulted  in  considerably  lower  hit  rates  for 
ground  and  air  targets  than  any  of  the  other  cueing  conditions  (Visual  [Vis],  Auditory  plus 
Visual[A+V],  Adaptive  Auditory  plus  Adaptive  Visual[Adap(A+V)],  Adaptive  Auditory  plus 
Visual[Adap(A)+V],  Adaptive  Visual  plus  Auditory[Adap(V)+A]),  which,  in  turn,  did  not  differ 
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among  themselves.  In  addition,  performance  in  the  unimodal  Auditory  condition  was  superior  to 
that  in  the  Non-Cueing  condition,  but  only  for  the  ground  targets,  which  were  more  difficult  to 
detect. 

The  relatively  poor  performance  for  the  Non-Cueing  condition  was  predicted,  since  in  that 
condition  pilots  were  not  provided  assistance  in  detecting  targets.  However,  the  limited  ability  of 
spatialized  auditory  cueing  to  aid  target  designation  by  itself  or  to  enhance  the  effectiveness  of 
visual  cueing  was  not  consistent  with  expectations,  which  were  based  upon  the  beneficial  role 
found  for  spatialized  auditory  cueing  in  other  investigations  (Perrott,  et  al.  1996;  Rudmann  & 
Strybel,  1999).  This  disparity  may  be  attributable,  in  part,  to  the  resolution  capabilities  of  the 
Convolvotron  system,  which  relied  on  generic  HRTFs  to  interpolate  source  locations  that  were  not 
among  the  72  locations  sampled  directly  by  the  system.  Indeed,  an  evaluation  conducted  in  the 
SIRE  facility  indicated  that  localization  error  for  the  Convolvotron  system  averaged  6  degrees 
under  optimal  conditions  (Moroney,  1999),  a  value  that  is  approximately  three  times  greater  than 
the  spatial  resolution  error  of  the  human  auditory  system  (Perrott,  et  al.  1990).  It  is  noteworthy 
that  previous  research  conducted  at  the  SIRE  facility  used  this  equipment  to  enhance 
performance  in  search  tasks  (Nelson,  et  al.  1998),  but  unlike  the  present  case,  the  tasks  did  not 
involve  simulated  flight  and  the  targets  to  be  detected  were  embedded  in  a  uniform  dark  field 
rather  then  the  realistic  scene  used  here  (desert  terrain  and  light  blue  sky). 

In  addition  to  the  spatial  resolution  problem,  it  may  be  important  that  the  pilots  in  this 
study  were  highly  skilled  in  the  use  of  visual  flight  instruments  including,  HDDs  and  HMDs.  Their 
familiarity  with  visual  displays,  coupled  with  their  relatively  limited  exposure  and  training  with  the 
auditory  cueing  system,  suggest  that  pilots  may  have  relied  mostly  on  the  visual  display  and  the 
search  behaviors  associated  with  that  display.  Consistent  with  this,  several  pilots,  on  debriefing, 
reported  that  they  utilized  auditory  cueing  to  alert  them  to  the  presence  of  a  target  and  to  the 
general  region  in  which  to  perform  a  visual  search,  but  not  for  precise  spatial  localization. 

Given  the  results  obtained  with  the  designation  accuracy  measure,  one  might  conclude 
that  the  presence  of  a  visual  cue  was  the  necessary  and  sufficient  factor  to  bolster  performance 
efficiency  in  this  study.  It  is  worth  noting,  however,  that  designation  accuracy  is  a  gross  measure 
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of  visual  search  performance.  Accordingly,  it  may  not  reflect  subtle  contributions  to  target  search 
made  by  the  integration  of  audio  and  visual  displays  which  might  appear  in  a  more  fine-grained 
speed  measure.  Such  contributions,  if  observable,  could  be  potentially  quite  important,  since  the 
time  to  locate  and  select  a  target  is  a  critical  factor  in  air  combat.  The  first  shot  advantage  greatly 
increases  the  probability  of  a  kill,  and  reduces  the  exposure  to  potential  threats  (Barnes,  1989; 
Shaw,  1988).  Consequently,  time  to  designate  targets  in  the  Visual  and  multimodal  cueing 
conditions  was  also  examined. 

Designation  Time.  Results  with  regard  to  designation  time  revealed  a  benefit  for  audio¬ 
visual  cueing  relative  to  visual  cueing  alone.  This  benefit  was  complex,  however,  involving 
interactions  among  target  type  (ground  or  air),  location  in  the  field  of  view  (FOV)  of  the  display, 
and  cueing  condition.  For  ground  targets  initially  outside  of  the  display  FOV,  designation  time  was 
approximately  825msec  slower  in  the  unimodal  visual  condition  than  in  the  fixed  and  adaptive 
multimodal  cueing  conditions.  This  effect  is  consistent  with  the  finding  of  reduced  detection  time 
for  targets  outside  the  FOV  previously  reported  with  the  use  of  spatialized  audio  cues  in  visual 
tasks  (Perrott,  et  al.  1990;  Perrott,  el  al.  1991).  The  benefit  of  audio-visual  cueing  was  not 
observed  for  ground  targets  initially  inside  the  display  FOV  nor  was  it  observable  for  air  targets 
either  inside  or  outside  the  display  FOV.  As  was  seen  in  the  designation  accuracy  data,  ground 
targets  outside  the  FOV  were  the  most  difficult  to  detect.  Consequently,  it  appears  that  the  benefit 
of  combining  spatialized  auditory  cueing  with  visual  cueing  existed  only  for  targets  that  were  the 
least  salient  and,  therefore,  where  supplementary  cueing  was  most  needed. 

The  time  differences  in  designating  air  versus  ground  targets,  and  targets  inside  the  FOV 
as  compared  to  those  outside  the  FOV,  also  warrant  comment.  The  significant  time  advantage 
noted  in  this  study  (>  1000msec)  for  air  over  ground  targets  may  be  due  to  the  relatively  low- 
contrast  between  the  ground  targets  and  the  terrain,  which  made  it  difficult  to  separate  target  from 
background  as  evidenced  by  the  lower  hit  rates  for  the  ground  as  compared  to  the  air  targets. 

The  finding  that  adding  spatialized  auditory  cues  to  a  visual  localization  interface  enhanced  the 
speed  of  ground  target  designation  supports  recent  research  demonstrating  the  utility  of  auditory 


35 


cueing  in  visually  demanding  search  tasks  (Bolia,  D’Angelo,  &  McKinley,  in  press;  Rudmann  & 
Strybel,  1999). 

The  designation  time  advantage  for  targets  initially  outside  of  the  FOV  over  those  within 
the  central  FOV  was  unexpected  given  the  higher  hit  rates  for  the  latter  than  the  former.  One 
possible  explanation  for  this  result  is  that  targets  initially  within  the  central  FOV  were  usually  in 
the  flight  path,  possibly  affording  pilots  more  time  to  designate  them  while  handling  other  flight 
demands.  In  contrast,  targets  outside  the  central  FOV  were  more  likely  to  be  off  the  flight-path 
axis,  which  meant  that  they  were  likely  to  disappear  quickly  from  the  OTW  view  (off  the  screen) 
and  needed  to  be  responded  to  immediately  or  be  lost.  It  is  unlikely,  however,  that  pilots 
intentionally  delayed  designating  targets  given  their  knowledge  that  speed  counts  in  combat. 

An  alternative  explanation  is  that  attention  to  the  HUD  and  HMD  may  have  reduced 
sensitivity  to  targets  initially  in  the  central  FOV.  Several  researchers  (Boston  &  Braun,  1996; 
Dudfield,  Hardiman,  &  Selcon,  1995;  Long  &  Wickens,  1994;  Martin-Emerson  &  Wickens,  1997; 
McCann  &  Foyle,  1994;  Ververs  &  Wickens,  1996;  Weintraub  &  Ensing,  1992)  have  addressed 
the  issue  of  having  to  switch  visual  attention  between  displayed  information  and  the  out-the- 
window  view  when  using  HUDs  and  HMDs.  Although  these  conformal  displays  eliminate  the 
need  to  physically  shift  one's  head  or  gaze  between  locations,  it  is  still  necessary  to  shift 
attention  between  the  display  and  the  outside  scene  (Boston  &  Braun,  1994).  Research  into  this 
phenomenon  has  demonstrated  that  when  symbology  overlaps  the  outside  visual  scene  it  can 
obscure  the  OTW  view  and  pull  a  pilot's  attention  away  from  that  scene  (“cognitive  fixation”).  As 
a  result,  responses  to  far-domain  events  are  delayed  (Long  &  Wickens,  1 994;  Martin-Emerson  & 
Wickens,  1997).  Additionally,  response  delay  for  events  in  the  far  domain  is  a  function  of  the 
density  of  events  in  the  near  domain:  the  greater  the  number  of  elements  in  the  near  domain,  the 
longer  the  response  time  to  events  in  the  far  domain  (Martin-Emerson  &  Wickens,  1997;  Ververs 
&  Wickens,  1996).  In  the  present  study,  targets  that  were  initially  in  the  central  FOV  tended  to  fall 
within  the  display  areas  of  both  the  HUD  and  the  HMD,  resulting  in  a  dense  visual  field.  In 
comparison,  targets  outside  of  the  FOV  appeared  in  a  less  dense  visual  field  since  they  were 
less  likely  to  be  overlapped  by  the  HUD  symbology. 
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Head  Motion 


Head  motion  is  another  important  aspect  of  visual  search  performance.  Excessive  head 
motion  can  lead  to  fatigue  and  reductions  in  performance  efficiency,  particularly  when  using  an 
HMD  in  high-g  environments  (Davis,  1 997).  This  measure  can  also  serve  as  a  general  indicator 
of  the  level  of  scanning  performed,  and  thereby  of  the  proportion  of  time  available  to  perform 
other  tasks.  Accordingly,  head  motion  in  this  study  was  examined  descriptively  by  means  of  the 
track  of  motion  in  azimuth  and  elevation  and  quantitatively  in  terms  of  mean  angular  head 
displacement.  Since  pilots  in  the  Non-Cueing  condition  were  not  alerted  to  the  presence  of 
targets,  one  might  expect  this  condition  to  necessitate  constant  visual  scanning  for  targets, 
producing  frequent  head  movements.  Consistent  with  such  an  expectation,  a  relatively  dense 
track  of  head  motion  and  a  relatively  high  amount  of  angular  head  displacement  occurred  in  the 
Non-Cueing  condition.  As  was  the  case  in  an  earlier  study  by  Nelson  et  al.  (1 998),  head  motion, 
both  in  terms  of  track  density  and  angular  displacement,  was  reduced  by  the  presence  of  a 
unimodal  spatialized  auditory  cue.  However,  the  reduction  of  head  motion  was  even  greater  in 
the  presence  of  a  unimodal  visual  cue  and  combining  audio-visual  cues  did  not  reduce  head 
motion  any  more  than  did  visual  cueing  alone.  Thus,  the  pattern  of  results  for  head  motion 
paralleled  that  for  detection  accuracy,  perhaps  because  of  the  limited  resolution  of  the  auditory 
cue  and  pilots’  predominant  experience  with  visual  displays. 

Flight  Performance  and  Workload 

One  of  the  risks  of  interface  development  in  a  complex  multitask  environment  is  that 
performance  improvement  on  one  task  may  come  at  the  expense  of  efficiency  on  other  tasks 
(Latorella,  1998;  Wickens,  1992).  Moreover,  the  means  used  to  enhance  performance  may 
induce  unacceptable  levels  of  workload  (Proctor  and  Van  Zandt,  1994).  To  assess  these 
possibilities,  the  influence  of  the  several  cueing  conditions  on  concurrent  flight  performance  was 
examined  along  with  the  subjective  workload  associated  with  the  cueing  conditions.  None  of  the 
interfaces  exceeded  the  Non-Cueing  condition  in  terms  of  deviations  from  the  prescribed  air 
speed,  flight  path,  and  altitude.  Thus,  the  enhanced  search  performance  produced  by  the 
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unimodal  visual  and  multimodal  conditions  did  not  come  at  a  cost  in  pilots’  ability  to  handle  other 
flight-related  activities.  Moreover,  the  workload  associated  with  the  visual  and  the  multimodal 
cueing  conditions  fell  at  a  level  (a  score  of  approximately  40  on  the  NASA-TLX)  that  is  considered 
acceptable  in  flight  simulation  tasks  (Moroney,  1999)  and  was  significantly  lower  that  that  in  the 
Non-Cueing  and  unimodal  Auditory  conditions  whose  means  (>  60)  fell  at  a  level  indicative  of  a 
need  for  reductions  in  workload  in  such  tasks. 

User  Feedback 

A  final  consideration  is  the  response  of  pilots  to  the  various  interface  configurations,  as 
the  pilots  are  the  true  end-users  and  experts.  User  feedback  and  acceptance  is  an  extremely 
important  aspect  of  interface  design  (Nielsen,  1993;  Shneiderman,  1992),  since  even  the  best 
designed  interface  is  of  no  value  if  people  chose  not  to  use  it.  In  this  experiment,  pilots  responded 
favorably  to  the  various  interfaces,  including  the  adaptive  configurations  in  which  they  were 
confronted  with  the  need  to  switch  between  auditory  and  visual  information.  As  indicated  above, 
pilots  reported  that  the  auditory  cue  was  useful  as  an  alerting  mechanism  to  the  presence  of  a 
target.  They  also  suggested  that  it  would  be  even  more  valuable  if  localization  resolution  could  be 
improved  so  that  head  motion  would  not  be  needed  to  precisely  determine  the  position  of  a 
target.  In  addition,  the  pilots  indicated  the  transition  between  audio  and  visual  information  in  the 
adaptive  conditions  was  easy  to  accomplish. 

Conclusions 

The  first  goal  of  this  research  was  to  determine  whether  multimodal  localization  cueing 
would  aid  in  visual  search  performance  when  using  a  simulated  HMD.  The  expected  benefits  of 
auditory  cueing  in  conjunction  with  visual  cueing  were  not  present  in  regard  to  the  accuracy  of 
target  designation.  At  first  glance,  a  result  of  this  sort  would  seem  to  support  the  findings  of 
Flanagan  et  al.  (1998),  who  reported  that  visual  search  performance  with  an  opaque  HMD  is  not 
aided  by  the  addition  of  spatialized  auditory  cueing.  On  the  other  hand,  the  present  study 
demonstrated  that  when  the  speed  of  target  designation  is  considered,  the  addition  of  spatialized 
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sound  to  a  visual  cue  does  enhance  performance  efficiency  in  comparison  to  the  visual  cue  alone 
under  conditions  in  which  targets  are  most  difficult  to  detect.  This  finding  is  of  considerable 
practical  significance  since  even  slight  time  advantages  can  be  critical  in  the  domain  of  tactical 
aviation.  In  addition,  multimodal  cueing  had  the  added  benefits  of  reducing  excessive  head 
motion  and  lowering  pilots’  workload  in  comparison  to  a  case  in  which  no  cueing  was  available. 
Moreover,  the  benefits  associated  with  multimodal  cueing  did  not  incur  a  performance-resource 
debt  (cf.  Wickens,  1992),  in  which  performance  on  concurrent  flight  tasks  was  sacrificed  for 
enhancement  in  the  speed  of  target  detection.  In  sum,  the  present  study  suggests  that 
multimodal  cues  may  effectively  aid  target  localization  when  using  a  see-through  HMD  in  tactical 
aviation. 

A  second  goal  for  this  study  was  to  explore  the  potential  advantages  of  the  adaptive 
presentation  of  multimodal  cueing  information.  Unlike  previous  studies  by  Brickman  et  al.  (1996) 
and  Moroney  (1999),  the  present  investigation  revealed  no  advantage  for  presenting  multimodal 
information  adaptively  over  presenting  it  in  a  fixed  format;  that  is,  the  benefits  associated  with 
multimodal  information  were  identical  in  both  formats  since  the  scores  for  the  Auditory  plus  Visual 
[A+V]  condition  were  similar  to  those  of  the  Adaptive  Auditory  plus  Adaptive  Visual  [Adapt  (A+V)], 
Adaptive  Auditory  plus  Visual  [Adapt  (A)  +V],  and  Adaptive  Visual  plus  Auditory  [Adap  (V)  +  A} 
conditions  with  respect  to  the  speed  and  accuracy  of  target  detection  and  reductions  in  head 
motion  and  workload.  The  failure  of  the  adaptive  conditions  to  prevail  over  the  fixed  format 
conditions  in  this  study  may  stem  from  the  fact  that  the  adaptive  format  was  not  necessary  in  the 
flight  scenario  used  here.  Recall  that  the  purpose  of  the  adaptive  format  was  to  reduce  cockpit 
clutter.  Since  it  turned  out  that  the  fixed  multimodal  interface  did  not  interfere  with  the 
performance  of  the  concomitant  flight  tasks,  there  was  no  added  benefit  in  clutter  reduction  to  be 
gained  through  the  adaptive  format. 

Implications  for  Future  Research 

The  results  of  this  research  suggest  several  avenues  for  further  study.  It  would  be  of 
value  to  determine  whether  the  difficulty-specific  advantage  found  for  multimodal  cueing  is  also 
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present  in  other  visually  restricted  search  tasks  such  as  those  performed  in  night-vision  and 
inclement  weather  conditions,  as  well  as  in  low-resolution  virtual  environments. 

One  feature  of  the  present  study  that  could  be  improved  upon  is  the  presentation  of 
spatialized  auditory  cues.  Clearly,  future  research  should  examine  the  efficacy  of  auditory  cueing 
with  conformal  visual  displays  using  a  higher  resolution  auditory  system  than  that  used  herein. 
Also,  the  utility  of  multimodal  localization  interfaces  might  be  revealed  more  effectively  when 
pilots  are  exposed  to  such  interfaces  early  in  their  flight  training  rather  than,  as  in  the  present 
case,  testing  them  after  they  have  had  substantial  experience  in  relying  solely  on  visual  displays. 
Extensive  training  with  auditory  displays  and  virtual  sound  localization  tasks  may  also  aid 
seasoned  pilots  in  using  multimodal  interfaces 

The  usefulness  of  an  adaptive  format  in  reducing  the  demands  of  cockpit  clutter  when 
presenting  multimodal  information  about  target  localization  might  emerge  in  a  flight  scenario 
requiring  greater  attention  to  displays  of  near  domain  information  then  what  was  called  for  in  this 
study.  Near-domain  load  could  be  increased  by  adding  additional  head-down  displays  and  their 
related  tasks.  These  could  include  systems  management,  communication,  and  mission  planning 
procedures. 

The  unexpected  finding  of  a  time  disadvantage  in  designating  targets  initially  within  the 
display  field-of-view  might  also  be  addressed  by  an  adaptive  interface  approach.  More 
specifically,  it  might  be  useful  to  employ  an  adaptive  procedure  to  remove  non-critical  conformal 
display  elements  when  a  target  is  present  in  the  display  field  of  view,  or  even  to  temporarily  shift 
some  of  those  elements  to  an  auditory  display. 
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APPENDIX  A 


Analysis  Summary  Tables 


Table  A1 

Analysis  of  Variance  for  Target  Designation  Accuracy. 


Source 

df 

df  (adj.) 

MS 

F 

P 

Target  (A) 

1 

27.82 

463.67 

<.05 

FOV  (B) 

1 

8.99 

99.89 

<.05 

Interface  (C) 

6 

3 

6.78 

61.64 

<.05 

AB 

1 

0.11 

1.57 

n.s. 

AC 

6 

3 

0.86 

17.20 

<.05 

BC 

6 

3 

0.18 

1.80 

n.s. 

ABC 

6 

3 

0.08 

1.14 

n.s. 

Subjects  (S) 

11 

0.08 

SA 

11 

0.06 

SB 

11 

0.09 

SC 

66 

40 

0.11 

SAB 

11 

0.07 

SAC 

66 

40 

0.05 

SBC 

66 

40 

0.10 

SABC 

66 

40 

0.07 

Table  A2 

Analysis  of  Variance  for  Ground  Target  Designation  Accuracy. 

Source 

df 

df  (adj.) 

MS 

F 

£ 

Interface  (A) 

6 

3 

3.10 

77.50 

<.05 

Subjects  (S) 

11 

0.05 

SA 

66 

40 

0.04 

Table  A3 

Analysis  of  Variance  for  Air  Target  Designation  Accuracy. 

Source 

df 

df  (adj.) 

MS 

F 

P 

Interface  (A) 

6 

3 

0.72 

18.00 

<.05 

Subjects  (S) 

11 

0.02 

SA 

66 

40 

0.04 
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Table  A4 

Analysis  of  Variance  for  Designation  Time. 


Source 

df 

df  (adj.) 

MS 

F 

P 

Target  (A) 

1 

116.31 

969.25 

<.05 

FOV  (B) 

1 

1.99 

12.44 

<.05 

Interface  (C) 

4 

2 

.99 

5.21 

<.05 

AB 

1 

.34 

6.80 

<.05 

AC 

4 

2 

.81 

7.36 

<.05 

BC 

4 

2 

.436 

4.36 

<.05 

ABC 

4 

2 

.685 

6.85 

<.05 

Subjects  (S) 

11 

.14 

SA 

11 

.12 

SB 

11 

.16 

SC 

44 

20 

.19 

SAB 

11 

.05 

SAC 

44 

20 

.11 

SBC 

44 

20 

.10 

SABC 

44 

20 

.10 

Table  A5 

Analysis  of  Variance  for  Angular  Head  Displacement. 

Source 

df 

df  (adj.) 

MS 

F 

P 

Interface  (A) 

6 

2 

9041324.27 

187.03 

<.05 

Subjects  (S) 

10 

1510184.49 

SA 

60 

20 

48342.329 

Table  A6 

Analysis  of  Variance  for  RMS  Airspeed  Error. 

Source 

df 

df  (adj.) 

MS 

F 

2 

Interface  (A) 

6 

2 

51.04 

3.00  1 

n.s. 

Subjects  (S) 

11 

52.11 

SA 

66 

20 

17.03 
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Table  A7 

Analysis  of  Variance  for  RMS  Lateral  Error. 


Source 

df 

df  (adj.) 

MS 

F 

P 

Interface  (A) 

6 

3 

265006.90 

4.32 

<.05 

Subjects  (S) 

11 

100977.98 

SA 

66 

30 

61400.32 

Table  A8 

Analysis  of  Variance  for  RMS  Above-Ground-Level  Error. 

Source 

df 

df  (adj.) 

MS 

F 

P 

Interface  (A) 

6 

4 

1921.93 

3.11 

<.05 

Subjects  (S) 

11 

3544.58 

SA 

66 

40 

617.95 

Table  A9 

Analysis  of  Variance  for  Global  Workload  Ratings. 

Source 

df 

df  (adj.) 

MS 

F 

P 

Interface  (A) 

6 

3 

1483.08  12.80 

<.05 

Subjects  (S) 

11 

943.31 

SA 

66 

30 

115.83 
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APPENDIX  B 


List  of  Abbreviations  and  Acronyms 


Interfaces 

Non 

Non-cueing 

Aud 

Auditory 

Vis 

Visual 

A+V 

Auditory  plus  Visual 

Adap(A+V) 

Adaptive  Auditory  plus  Adaptive  Visual 

Adap(A)+V 

Adaptive  Auditory  plus  Visual 

Adap(V)+A 

Adaptive  Visual  plus  Auditory 

Acronyms 

AGL 

Above-ground-level  altitude 

FOR 

Field-of-regard 

FOV 

Field-of-view 

HMD 

Helmet-mounted  display 

HMS 

Helmet-mounted  sight 

HOBS 

High  off-axis  boresight  weapons  system 

HRTF 

Head-related  transfer  function 

HSD 

Horizontal  situation  display 

HUD 

Head-up  display 

LOS 

Line-of-sight 

OTW 

Out-the-window 

RMS 

Root  mean  square 

SIRE 

Synthesized  Immersion  Research  Environment 
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